Papers with subword tokenization methods
Revisiting subword tokenization: A case study on affixal negation in large language models (2024.naacl-long)
Copied to clipboard
| Challenge: | Negation is central to language understanding but is not properly captured by modern NLP methods. |
| Approach: | They propose to use subword tokenization methods to detect negation in large language models . they find that models can reliably recognize negation, despite mismatches in tokenization accuracy . |
| Outcome: | The proposed models can detect negation in English using subword tokenization methods despite some mismatches in tokenization accuracy and negation detection performance. |
Byte Pair Encoding is Suboptimal for Language Model Pretraining (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Subword tokenization is a popular language model that can be used to segment text. |
| Approach: | They analyze differences between byte-pair encoding (BPE) and unigram LM tokenization methods to find subword units that align more closely with morphology. |
| Outcome: | The proposed method recovers subword units that align more closely with morphology and avoids problems stemming from BPE’s greedy construction procedure. |